In order to group countries based on their similarity, we decided to plot a dendrogram based on the correlation matrix of those countries.
The dendrogram illustrates how each cluster is composed by drawing a U-shaped link between a non-singleton cluster and its children.
- The top of the U-link indicates a cluster merge.
- The two legs of the U-link indicate which clusters were merged.
- The length of the two legs of the U-link represents the distance between the child clusters.
- It is also the cophenetic distance between original observations in the two children clusters.
Initially, before starting the algorithm, each feature is a cluster. The scipy.linkage library uses what is called 'agglomerative clustering'.
- It starts with each observation as its own cluster.
- The two closest clusters are joined into one cluster.
- The next closest clusters are grouped together
- GOTO 1: this process continues until there is only one cluster containing the entire data set.
Each leaf in the dendrogram represents a feature and each node a cluster. The y-axis shows the distance between points.
The number of clusters in our data will depend on which distance we take as a threshold. If we select a small distance, more clusters will be formed. Conversely, if we choose a large distance as a threshold, we would less clusters.
The most popular ways to define distance between clusters are the following methods:
Simple linkage: Single-linkage (nearest neighbor) is the shortest distance between a pair of observations in two clusters. It can sometimes produce clusters where observations in different clusters are closer together than to observations within their own clusters. These clusters can appear spread-out.
- Problem: it tends to be extremely myopic. An object will be added to a cluster so long as it is close to any one of the other objects in the cluster, even if it is relatively far from all the others. Thus, single linkage has a tendency to produce long, stringy clusters and nonconvex cluster shapes.
Complete linkage: Complete-linkage (farthest neighbor) is where distance is measured between the farthest pair of observations in two clusters. This method usually produces tighter clusters than single-linkage, but these tight clusters can end up very close together. Along with average-linkage, it is one of the more popular distance metrics.
- Problem: Can be highly sensitive to outliers in the data.
- Average linkage: Average-linkage is where the distance between each pair of observations in each cluster are added up and divided by the number of pairs to get an average inter-cluster distance. Compromise between single and complete linkage
- Ward’s method: seeks to join the two clusters whose merger leads to the smallest within-cluster sum of squares (i.e. minimum within-group variance). It has a tendency to produce equal-sized clusters that are convex and compact
https://stats.stackexchange.com/questions/195456/how-to-select-a-clustering-method-how-to-validate-a-cluster-solution-to-warran/195481#195481
Average:
https://www.researchgate.net/publication/339443595_Hierarchical_Cluster_Analysis_Comparison_of_Single_linkageComplete_linkage_Average_linkage_and_Centroid_Linkage_Method
Average:
https://journalofinequalitiesandapplications.springeropen.com/articles/10.1186/1029-242X-2013-203
Ward:
https://people.stat.sc.edu/Hitchcock/compare_hier_fda.pdf
It was decided that wards method was the best option (https://people.stat.sc.edu/Hitchcock/compare_hier_fda.pdf) and provided a more reasonable and intuititive clustering. Despite that, we observe that the first (and most important) grouping is identical to all linkage methods, and the differences
The dendrogram that is plotted can be used to group countries based on the similarity of their datasets for later transfer learning
Since dendrogram is plotted based on correlation of countries, that means that it is not affected by the scale or distribution of countries' data. To have a more complete criterion for grouping, we use heatmaps based on countries and time covariates which is affected by the scale/distribution of data
Ireland/Portugal happens to be a single cluster, but obviously a small one. By observing the correlation matrix we can, with great confidence, group them with the purple cluster, since Ireland mostly correlates with those countries (e.g Spain)
As further observation we see three (3) major groupings, going from left to right these clusters are mostly populated by:
- northern countries (Norway/Sweden/Switcherland)
- countries of central Europre
- countries near Meditterenean
An obvious reason would be the different metereological conditions and needs for energy demand based on that
Despite the difference in each country's domain (distribution), this is hardly a setback to our transfer learning practice by using what is called domain adaptation: a subcategory of transfer learning in which, the source and target domains all have the same feature space (datetime), but different distributions (load)
